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Abstract 



The storage capacity of the extremely diluted Hopfield Model is studied by 
using Monte Carlo techniques. In this work, instead of diluting the synapses 
according to a given distribution, the dilution of the synapses is obtained 
systematically by retaining only the synapses with dominant contributions. 
It is observed that by using the prescribed dilution method the critical storage 
capacity of the system increases with decreasing number of synapses per 
neuron reaching almost the value obtained from mean-field calculations. It 
is also shown that the increase of the storage capacity of the diluted system 
depends on the storage capacity of the fully connected Hopfield Model and 
the fraction of the diluted synapses. 
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1 Introduction 



Statistical mechanics provides very powerful theoretical framework to study 
the storage capacities and the generalization abilities for a variety of artificial 
neural network architectures. One of the most popular and well-studied arti- 
ficial neural network model is the Hopfield Model. In the Hopfield Model^], 
neurons have two levels of activity and in the original form of model, the 
neurons are connected with all other neurons with synaptic couplings. The 
synaptic couplings, Jjj, between i^^ and neurons are set by the Hebb 
Learning Process [2] which ensures the dynamic stability of the initially in- 
troduced patterns. The pattern in the original set of patterns, are 
assumed to be random with equal probabilities for = ±1. 

One can define the critical storage capacity (oc) as. 



where N is the number of neurons and Pmax is the maximum number of 
patterns that can be stored into the neuron system before the total loss of 
memory The critical storage capacity {a^ for the two-state fully con- 
nected neuron system is calculated by means of mean-field and numerical 
simulation methods. Typical values are 0.1379Pj and 0.144j3] by using 
replica symmetry breaking method and 0.144 ± 0.009 3 , 0.145 ± 0.001|5] 
and 0.143 ± 0.001 6 by using numerical simulation techniques. 

For the fully connected networks the critical storage capacity sets severe 
restrictions on the applicability of the model to more realistic problems of 
the physical systems. In this case one turns to the biological systems where 
the storage capacity is extremely higher than the above given values. In the 
biological neural networks, the neurons are very sparsely connected and the 
connections are set by the personal experiences. This extremely successful 
example led scientists to devote a special interest to the diluted networks 
where only a fraction of neurons is connected. Theoretical studies showed 
that the storage capacity per neuron in diluted artificial neuron systems 
can be substantially larger than the fully connected networksi?! |H]. This 
theoretical result can intuitively be understood by considering the local field 
acting on a single neuron. If the stored patterns are very limited in number, 
the effective local field sets the sign of the neuron (sign of the local field) 
at a given site without any ambiguity. Increasing the number of stored 
patterns causes frustration on the neurons. Hence dilution may decrease the 
number of frustrated neurons, therefore increase in the storage capacity may 
be observed. 

In this work we aim to study the storage capacity of the extremely diluted 
Hopfield Model by using Monte Carlo simulation techniques. We aim to 
introduce a dilution prescription which resembles the biological learning pro- 
cess. In the selection of the remaining connections the most prominent con- 
tributions to the connection matrix are considered for each neuron. This 
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prescription leads to an extremely diluted system and yet the connections 
are still sufficient to recover large number of stored patterns. 

The paper is organized as follows: In the next section we have introduced 
our dilution method and the thermodynamic observables in order to study 
memory effects. In the third section we have presented our simulation results 
and how the storage capacity of the diluted Hopfield network can change by 
using our dilution techniques. The final section is devoted to the discussion 
of our results and conclusion. 

2 Model and the Method 

The dynamics of the two-state neuron system is defined by the Hopfield 
Hamiltonian; 

-H" = - XI Jij^i^j (2) 

where Si is the i*'* binary neuron that can take the values ±1. We have used 
zero temperature dynamics to update the neurons at a given time t according 
to the following rule; 

Si{t + l)^Sign{Y,JijSM (3) 

and the synaptic matrix element between i*^ and j*^ neurons, Jy, is defined 
as; 

Jij = a, E (4) 

where P indicates the number of initial patterns to be stored in the system, 
i and j run from 1 to A'^ and N is the number of neurons in the system. 
The elements of the dilution matrix, C, define the dilution of the synapses. 
The Cjj's can take the value "1" or "0" depending on whether the connection 
between i*'* and neuron exits or not. 

In our study, the dihition of the couplings is realized in a guided way. Instead 
of diluting the connections according to a given distribution, we have intro- 
duced a prescription by which the dilution of the synapses is realized. Our 
prescription resembles very much the actual learning process: In the learning 
process some events or patterns are easily recalled if many different events 
or patterns in the past experiences have similarities or relations. In fact in 
the fully connected Hopfield Model, since the original patterns have random 
orientations at the j^^ position, the coupling matrix J may contain many 
zeros or very small random values whose contribution to the dynamics of the 
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i*'^ neuron is only to randomize the system at the following time steps. The 
most prominent contributions to the dynamics of the i^^ neuron are coming 
from the sites, j, at which many different original patterns posses the same 
value. The number of sites with this property rapidly decreases as the num- 
ber of initially thought patterns increases. Nevertheless, the elimination of 
the synapses with random contributions will emphasize the effect of these 
sites. These contributions are decisive contributions for the orientation of 
the i*^ neuron at the time t + 1 when the system is extremely diluted. In the 
case of extremely diluted Hopfield Model, many couplings will be taken as 
zero leaving only those that contribute the most. Since the majority of the 
neighboring neurons indicates a net orientation and hence in the summation 
in Eq. (0)) the cancellations are minimal. 

To implement this prescription, the strengths of the connections between i*'* 
neuron and all other neurons are calculated prior to the dilution process. In 
order to determine the undiluted synapses, we have considered how many 
neurons at a given site j in different patterns carry the same sign with the 
i*^ site, Tj*"™^, and how many neurons posses opposite sign, xhe 
number of like neurons in all patterns at the location j 



and the number of oppositely signed neurons at the location j, 



Tr''"=j:{i-6,^,^) (6) 

constitute two sets of values. These sets are sorted separately by value. Apart 
from the zero values which may exist in both sets the j values can not be the 
same. By merging these two sets together and by considering the fraction of 
the diluted neurons. Only the required number of terms starting from the 
highest values are taken into consideration. From this set one can obtain the 
locations (j's) of the neighboring neurons with non- vanishing synapses. The 
relevant elements of the dilution matrix C are set to unity starting from the 
largest value from both sets i.e. only the prominent contributions are taken 
into account. Even if the system is not extremely diluted or fully connected, 
the same prescription still works. 

In order to study the time evolution of a neuron system, the following ob- 
servables are calculated: 

i) The average overlap m{t) between the states S{t) and the /x*'* original 
pattern ^'^ that are all stored in the memory; 

1 P N 

^it) = ^T.T.^^m (7) 
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ii) The storage capacity of the system; 



P 

« = (8) 

' Synaps 

where Nsynaps is the number of remaining connections per neuron and P 
is the number of stored pattern in the diluted network. In this work we 
considered a fixed number of connections for each neuron. 



3 Results and Discussions 

In this work we have employed the zero temperature dynamics, Eq. Q, 
for the time evolution of the neurons. We have used four different-size neu- 
ron networks and for each one of these networks different dilution rates are 
employed. The number of neurons (A^) and the number of connections per 

neuron {Nsynaps) that are used in this simulation study are given in Table 1. 

In our studies the initial patterns are prepared for each size, iV, by using 
random linear distribution. The set of P initial patterns are stored into the 
system. The connections between the neurons are calculated by using the 
above given prescription until the required dilution is reached. The patterns 
are distorted by using 10% random noise. In our simulation work we have 
tested the recovery for each of the distorted initial patterns and this process 
is continued until all of the patterns are exhausted. The recovery rate is 
defined as the average of the overlaps between the original pattern and the 
final recovered pattern. 

In order to obtain the critical value of storage capacity, Uc-, we have started 
with very small number of stored patterns to our simulation process and we 
have increased the number of initial patterns for each N and Nsynaps until 
no more patterns can be recovered. This process of feeding the system with 
increasing number of initial patterns is continued while 100% overlap between 
the final patterns and any one of the original patterns is obtained. 

In Fig.l we have presented the storage capacity (a = — ) versus the 

Synaps 

number of recovered patterns for = 500 neuron system. Starting from the 
fully connected system {Nsynaps = 499), the behaviour of the diluted system 
with varying number of connections per neuron {Nsynaps = 300, 200, 100, 50) 
is presented. For a given number of synapses the number of stored patterns 
exhibits a peak. This peak becomes wide Gaussian-like and the maxima of 
the peaks are shifted toward the larger a values as the number of connections 
is reduced. The initial increase is linear and the slope of the linear initial 
increase decreases as the number of synapses decreases. The maximum num- 
ber of stored patterns also decreases but the number of stored patterns per 
synapse {P/Nsynaps) increases with decreasing number of synapses. There is 
a peak at a given a value and number of recovered patterns decreases as a 
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increases for each Nsynaps- We have apphed the hncar regression technique 
to the decreasing side of each Gaussian-hke curves to determine the critical 
storage capacities of the network for different number of synapses per neuron. 
These calculated storage capacities are used to determine the critical storage 
capacity for the fully connected and extremely diluted systems. These values 
are presented in Fig.4. 

In Fig. 2, we have presented the storage capacity (a) versus the average over- 
lap (m) between initial patterns and the recovered memory for a network 
of 500 neurons. Different curves represent different Ngynaps values. While 
the number of synapses decreases {Nsynaps = 250, 100, 50, 25, 20, 15, 6) the 
storage capacity increases. As it can be seen from Fig.2, for Nsynaps = 250, 
the storage capacity is already approximately twice the value of 0.14. As the 
number of synapses is decreased to Nsynaps = 6 this storage capacity reaches 
to an average value of 1.35. For a system with 500 neurons 6 synapses per 
neuron seem to be the limit of the possible dilution. 

In Fig. 3, we have presented the storage capacity (a) versus the overlap (m) 
between initial patterns and the recovered memory for the network of iV = 
400, 500 and 800 neurons. Here we have plotted Nsynaps = 160, 50 and 6 
connections per neuron for each different-size network. This figure shows the 
relation between the dilution rate and finite-size effects. The size dependence 
of the storage capacity diminishes as the dilution is increased. As the system 
becomes extremely diluted, the different-size networks yield the same storage 
capacity. In our case (N = 400,500,800), different network sizes yield the 
same critical storage capacity within the error bars which indicate that all 
curves follows the same path from the storage to the total memory loss phase. 

In order to study the limiting cases, we have presented the relation between 
the fraction of diluted synapses and the critical storage capacity. In two 
extreme regions we have fitted the data to the same form, 

aJFd) = . (9) 

where Fd is the fraction of diluted synapses. Fa = 1 — and ac(Fd) is 

the critical storage capacity at given dihition fraction. In Fig.4, the dihition 
fraction versus critical storage capacity is presented for the networks N = 
400, 500, and 800. The data and the fits to the data for the largest lattice, 
N — 800 at two extreme limits {Fd and — > 1) are also presented in this 
figure. In both limits the same functional form for the fit (Eq. 9) has been 
used. In the first limit (fully connected system) fit yields the Hopfield model 
critical storage capacity value as expected. The second region {Fd — * 1) 
includes data of at least 90% diluted system as seen in Fig.4. Fit to the data 
of the diluted synapse ratio Fd > 0.9 gives the critical storage capacity which 
is approximately equal to ac{l) = 1.6. 
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4 Conclusion 



In this work we have studied the storage capacity of extremely diluted Hop- 
field Model. Our dilution algorithm is based on a prescription which retains 
maximum information on the non-diluted connections. This is achieved by 
considering the sites which will be connected to the originally chosen neurons 
as the ones with the highest number of patterns containing like and unlike 
neurons. This choice ensures the sign of the i*^ neuron for the next time 
step. Decreasing number of connections increases the storage capacity of the 
system. In our study we have calculated the critical storage capacity of the 
extremely diluted Hopficld Model as about 1.6. 

With the well-defined dilution prescription given above and the rate of in- 
creased storage capacity, our method of dilution open possibilities for many 
applications. Particularly at the 100% recovery region, a large amount of 
information can be stored and recovered at a very low cost of memory space. 
Our studies in this direction is in progress. 
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Table captions 

Table 1 The size of neuron systems and the number of connections per neuron. 
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Figure captions 

Figure 1 The storage capacity (a) versus number of recovered patterns for 
Nsynaps — 500, 300, 200, 100, 50 and linear regressions. 

Figure 2 The storage capacity {a) versus average overlap (m) for 
Nsynaps = 250, 100, 50, 25, 20, 15, 6. 

Figure 3 The storage capacity (a) versus average overlap (m) for 
N = 400, 500, 800 and Nsy^aps = 160, 50, 6. 

Figure 4 The fraction of diluted connections (Ffi) versus the critical storage 
capacity (ac(^d)) for N = 400, 500, 800. 
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Network Size (N) Number of Couplings per Neurons (Nsynaps) 

400 400, 360, 320, 280, 240, 200, 160, 120, 100, 80, 50, 40, . .T76~ 

500 500, 450, 400, 350, 300, 250, 200, 160, 150, 100, 50, 25, . . . , 6 

800 800, 720, 640, 560, 480, 400, 320, 240, 160, 80, 60, 50, . . . , 6 



Table 1 
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